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ABSTRACT 


We live in a digital world and our students are surrounded by technologies: it is essential that they learn how to use the 
digital tools available for education purposes. As for language teaching, among the most recent technologies we can find 
online dictionaries. They could play a key role to foster language competences, and yet they are not very popular in schools. 
In order to increase the use of online dictionaries among students, the University of Turin has promoted the nation-wide 
project 'Esplorare (con) i Dizionari Digitali’. This project makes the most of a Learning Management System 
(https://esploradizionari.i-learn.unito.it/) that allowed the creation of a student-centered environment. The integration of 
the platform with an Automatic Assessment System allowed the design and creation of tests to be carried out through the 
consultation of online dictionaries. In this paper, we analyze the answers given by almost 600 students in 5 different tests 
on different languages, and we try to understand whether the methodologies and technologies adopted have been influential 
in making students "meta-linguistic aware" and researchers "meta-design aware". The results obtained show in a 
statistically significant way that the students who made a larger use of online dictionaries performed better in the tests. 
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1. INTRODUCTION 


Nowadays, students live in a technology-rich environment and quickly learn how to use different apps and 
technologies; their way of communicating with teachers and peers, interacting with learning materials, and 
demonstrating their knowledge have changed over the last years. For what concerns language teaching, among 
the various technologies that have spread since the advent of internet there are online dictionaries, 
i.e. monolingual or bilingual dictionaries (free or paid) that can be consulted via any mobile device. Online 
dictionaries could be a key tool to foster language competences, but they are not popular in schools. How can 
we motivate our students to use them? Teaching itself is not always enough, except for those languages 
(e.g. Latin) for which translation is still often practiced in class. The motivation should come directly from the 
layout of dictionaries, since they are always and easily available on a smartphone. One of the reasons why 
online dictionaries are not frequently used can be that they still need improvements in some of their search 
functions. At the same time, Italian lexicography market is not broad enough to justify a significant investment 
on it. Studying the use of digital dictionaries by students (Marello, 2014), we asked ourselves the following 
research question: how to increase the use and the circulation of online dictionaries between students? To 
answer to this question, two years ago the University of Turin has developed and promoted the nation-wide 
research project “Esplorare (con) 1 Dizionari Digitali”, that is “To Explore (with) Digital Dictionaries’, with 
the goal of helping students understand how online dictionaries are structured so that they can be consulted. 
The project makes the most of a Virtual Learning Environment (https://esploradizionari.i-learn.unito.it/) that 
allowed the creation of a student-centered environment, so that students can easily approach online dictionaries 
and be trained on their use. The integration of the platform with an Automatic Assessment System has allowed 
the creation of tests with exercises, integrated with the consultation of online dictionaries, grouped by difficulty 
levels that allow to verify lexical knowledge. The project was made possible thanks to a fruitful cooperation 
with some high school teachers, who voluntarily chose to participate with their classes. Our research group 
designed interactive exercises to stimulate the use of online dictionaries. Teachers gave us feedback on how to 
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refine them and chose which way to give the test to their students, whether at home or at school, in a 
computerized classroom supervised by teachers and researchers. 

In this paper, we analyze the answers given by almost 600 students in 5 different tests of different languages 
to understand whether the methodologies and technologies adopted have been influential in making students 
“meta-linguistic aware” and researchers “meta-design aware”. 


2. TECHNOLOGIES AND ONLINE DICTIONARIES TO FOSTER 
LANGUAGE COMPETENCES 


With the succession of new approaches and methods, suggested by the evolution of the psychology of education 
and theories of linguistic acquisition, but also by changing needs for new political, economic and social 
conditions, language teaching has resorted to various technological tools. Today, most linguists and language 
teachers consider unthinkable to teach and learn a language without using any technology (Betti & Garelli, 
2010). With the advent of the Internet (1989), computers have definitively been established as an unavoidable 
tool for teaching and learning languages, as they are able to offer authentic materials of various types that could 
be far more stimulating than those created ad hoc by teachers and authors that are more interested in grammar 
rules than in the context of use of the language. In addition to a new way of considering languages, seen as a 
tool to communicate in the most diverse situations and contexts, other circumstances have favored the use of 
the computer, such as the theory of social constructivism. According to this theory, a person, in his cognitive, 
social and affective aspects of behavior, is not a product of the environment, nor the simple result of his internal 
dispositions, but a personal construction that is produced, day by day, in the interaction of these two factors. 

The most significant reflection of constructivism in the teaching and learning of foreign languages is 
undoubtedly the recognition of the learner as an autonomous individual. The knowledge of a language is 
defined in terms of ability, of a "know-how", and the learner is considered as an autonomous and self-aware 
individual who works for the construction of his own future. Since 1976, the European Community has 
encouraged the use of technology in language teaching, implementing a real linguistic policy that emphasized 
the need to use multimedia, as recommended in the White Book of Education and Training (1995). More 
recently, in 2004-2006, the CoE drew up an Action Plan aimed at promoting language learning and linguistic 
diversity, thus giving everyone the opportunity to learn an L2 during the course of their lives, and improving 
their teaching through new technologies, as there are still teachers who are struggling to abandon traditional 
teaching techniques. Both teachers and students need to be taught how to efficiently use existing online 
resources, and students should be trained to a constant and more proficient autonomous use of such tools 
through specific tasks, so that they can expand their knowledge. 

Online dictionaries are a key tool to learn new languages and to deepen the knowledge of one’s mother 
tongue. Unfortunately, their use is not so widespread, as reports by the researchers cooperating in the European 
Network of e-Lexicography (ENeL) and in the European survey of dictionary use have shown that, in many 
countries, dictionaries and dictionary use have even been excluded from the curricula of high schools (Kosem 
et al., 2019). The cause of it could be seen in their design or their cost, which make them not accessible to 
everyone. In order to make them an exploited tool by students, it would be important to improve their layout, 
and to make them so useful, appealing, intuitive and self-explanatory that no training on their use should be 
necessary (Carr, 1997; De Schryver & Prinsloo, 2001; Klosa & Miiller-Spitzer, 2018; Lew, 2011; Lew, 2015; 
Tarp, 2012). 

The English Vocabulary Profile EVP is perhaps one of the closest achievements to this idea. It offers 
reliable information about which words (and importantly, which meanings of those words) and phrases should 
be known and used by learners at each level of the Common European Framework (CEF). There are the British 
and the American English versions, and an audio pronunciation for all entries. It is quite appealing, and it is 
designed for smartphone. However, experiments performed with university students demonstrate that they tend 
to use even the best digital dictionaries as if they were traditional dictionaries, which means they choose the 
first meaning listed, without taking advantage of all the features available within such dictionaries 
(Dziemianko, 2012). 

Freely available Italian online dictionaries might be largely improved; just to mention three easy ways to 
make them more user-friendly, we might deal with display of homonyms, a ‘did-you-mean’ function, 
somewhat like a spellchecker (Lew & Mitton, 2013) and a lemmatizer. The visualization of homonyms should 
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be improved, for example, typing “schifo” in the very good monolingual De Mauro on line it is not immediately 
clear that this word has two entries, the first one that means ‘small rowing boat” and the second one (which is 
actually the most common one) that means “sense of nausea, disgust”. The numbers used to distinguish them 
are so small that they are difficult to notice. Italian online dictionaries do not feature a ‘did-you-mean’ function 
“guessing” the correct spelling of words misspelled by users. In Garzanti, if you type the non existing entry 
coglio, they suggest ciglio, caglio, cogli; but if you type simbol instead of simbolo (i.e. symbol) no results are 
shown. Finally, a powerful lemmatizer is important above all in order to forward from inflected forms to the 
verb in its base form under which it is lemmatized and explained. 

The starting point to improve dictionaries is to know which aspects of the dictionary microstructure are the 
most difficult, according to the different types of users. Eye-tracking studies and log files analysis were carried 
out to achieve this goal (Lew, 2015; Tépel, 2014; section V in Gouws et al., 2013), but in Italy these techniques 
are still to be applied in the field of online lexicography. In fact, the Italian dictionary market is not sizeable 
enough to justify a significant investment by publishing companies to further improve the online layout of 
dictionaries and to fund experimental research on online testing. The consequence is that Italian university 
researchers and teachers associations are taking the initiative to address the gap in their skills, to try to make 
the most of the tools that already exist and to train students to use online dictionaries as they are at present 
(Marello, 2014). 


3. METHODOLOGY OF EXERCISE DELIVERY 


The groundwork of the project “Esplorare (con) i Dizionari Digitali” has been influenced by the experience of 
our research group in the use of an Automatic Assessment System (AAS) in order to develop skills and 
competences of STEM disciplines (Science, Technology, Engineering, Mathematics). Moodle, a Virtual 
Learning Environment (VLE), was used, integrated with the AAS Mobius Assessment (Barana et al., 2015). 
This integration has proven to be effective for the teaching of Mathematics (Barana et al., 2019b) and in open 
online courses (Marchisio et al., 2019a; Marchisio et al., 2019b) as it exploits the mathematical engine behind 
it: the Advanced Computing Environment (ACE) Maple. It also allowed the creation of a model of formative 
automatic assessment and interactive feedback for STEM (Barana et al., 2018). This is why the same 
technologies and methodologies were adopted, at first, for an experimentation in the creation of new typologies 
of language questions (Barana et al., 2019a) and then for the realization of the project “Esplorare (con) i 
Dizionari Digitali”. In 2016, a pilot project about the use of online dictionaries started; it used paper 
questionnaires while the use of computers and smartphones was limited to the consultation of online 
dictionaries. In 2017, we decided to improve the project with the use of the VLE and the AAS in order to allow 
the development of more exercises, to reach more students, and to better record their answers. The use of a 
VLE also allows to take more into consideration attitudes and needs of individuals and to foster the active 
learning (Felder & Brent, 2009). 

The e-learning Moodle platform we opened for the project has a collection of exercises in monolingual and 
bilingual dictionaries accessible through the AAS, which is available also for tablets and mobile devices. The 
platform is divided into three area. The first one is open and it can be accessed freely through the credentials 
of a social network. It contains a demo to learn how monolingual and bilingual online digital dictionaries are 
structured. The second area is where each teacher has their course dedicated to their class, with tests available 
for students. The third section is a teacher community where they can exchange best practices, discuss, create 
materials and receive support from university researchers. 

One of the main goals of the project is to train students to access online dictionaries and learn how to get 
information from them. We focused on maintaining a certain amount of exercises that foster skills and are not 
merely search options in free online tools, which are severely limited. This was particularly important, as 
teachers tend to favor lexical or reading comprehension exercises that reveal very little to nothing about the 
students’ practical skills in using and understanding dictionaries and the data they contain. To understand if 
students used online dictionaries and how they used them, at the beginning of our project we were always 
asking students at the end of each question which parts of the dictionary, if any, they used in order to solve 
some tasks we assigned through the platform. We collected many answers to these questions, but some turned 
out to be inconclusive. In fact, from the execution time recorded it was evident that some students did not use 
a dictionary at all, but they declared they had, probably only to please the researchers. In some other cases, 
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they affirmed they had not used it in order to impress their school teacher and to show that they already knew 
the correct answer without consulting the dictionary. After these results, we decided not to ask such 
metacognitive questions and we prepared other exercises instead, which could not be correctly carried out 
without knowing how to use dictionary microstructure. In order to increase the use and the circulation of online 
dictionaries among students, we found some search features that we thought important to make students work 
on and we designed exercises which make them learn how to overcome them. So far, “Esplorare (con) i 
Dizionari Digitali” has seen two editions, involving in total more than 600 students, mainly from Piemonte but 
also from Puglia and Trentino-Alto Adige (Table 1 reports all the numbers of the project). All the classes 
involved belonged to the upper secondary school, but in the 17/18 edition also undergraduate and postgraduate 
students were involved. Results present in this paper belong to both edition of the project while the 19/20 
edition will start soon. 


Table 1. Numbers of “Esplorare (con) i Dizionari Digitali” 


2017/2018 edition 2018/2019 edition 
Classes 9 (+ 3 university courses) 15 
Students 207 (+ 305 university students) 335 
Teachers 8 16 
# exercises carried out 125 200 


Each teacher can select the set of exercises among our collections in order to fit students’ needs and choose 
whether to have them do the exercises in the classroom or at home (according to the “flipped classroom” 
methodology), entering the activity in the normal school planning. Tests are set in a slightly different way 
according to this decision. When the test was carried out in class, teachers and researchers were checking on 
students in order to avoid cheating behavior and try to keep students focused on the test without distractions, 
without any time or attempts restrictions. When the test was carried out at home, they had an attempt limit 
(i.e. each student could attempt the test only once) and a time limit; in this way, students could choose when 
to do the test and hopefully they did it only when they had time to properly focus on it. 

The VLE and the AAS allow to record a lot of information about users’ logs, as well as the time and attempts 
for each test, and they automatically compute statistical data about the success rate of each question. Moreover, 
the AAS records all answers given by students and it allows their automatic evaluation with the exception of 
open answers (which were rarely used for this project). The AAS provides some tools that are useful after the 
tests, but also other tools that are quite important for their very creation. AAS gives the possibility to choose 
among many kinds of response-areas, and to set different options according to the goal of the exercise. 
Therefore, the AAS allowed to only use automated procedures for both parts of the study (data collection and 
evaluation). In addition, SPSS, a software package for batched statistical analysis, was used for most part of 
the analysis of results (explained in the following chapter). 


4. RESULTS 


4.1 When the Researcher Becomes “Design Aware” 


The AAS we used had several advantages: the main one was recording all answers given. This allows the 
designer to understand what students exactly did and to learn from results. Accessing online dictionaries is 
facilitated through the AAS: each question that requires the use of a dictionary has a specific pop-up link that 
directly sends the user to the dictionary entry to be consulted, and this speeds up the process, if compared to 
having the general dictionary consultation window. In addition, also the pop-up link to the consultation window 
speeds the process. To use pop-up links through Moodle, it was not possible to use the appropriate option to 
insert pop-ups designed by the creators of the AAS; it was necessary to design a special way in order to make 
it possible. Links were inserted as html-type answers to which students must not give an answer: the first time 
this was tested, the AAS saw the pop-up links as answers left empty, but we solved this problem by inserting 
the code “function getResponse() { return 1; };”. The html solution could still be further improved as the link 
disappears in the pdf version of the test and the pop-up poses (small) problems in the layout of the exercise, 
because it tends to create too much space between the parts of the question; these aspects are still under study. 
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The use of the AAS allows to set some options useful to control the way in which the test is done. Two 
options we set in every test were the rotation of questions inside the test and the rotation of possible answers 
to choose from: these decrease the probability of cheating, especially when students take the test in class. In 
some cases, to reduce the probability of cheating even more, we fully exploited the possibilities of the AAS of 
using algorithms: thanks to the command “switch”, we created three or four equivalent tests with different 
lemmas. Fig. | shows all the features listed above, and it is a pop-up case with the use of both the English 
dictionary (Collins) and the American one (Heritage). In this question, students have to look up in the online 
dictionary the plural forms of nouns written in red and select the correct form. For each attempt, nouns in red 
are different and the related possible answers rotate. 


Some English words have irregular plural forms and some have more 
than one possible plural form. Look up the plural forms for the following 
nouns and identify the correct form. Use the Collins English Dictionary , 
which will give you results from three different dictionaries, and the 
American Heritage Dictionary and only consider the sense referring to an 


animal. 


Mouses 


Both deer and deers Mouse 


Figure 1. English and American dictionaries pop-up links in AAS question 


Each test consisted of 7-10 questions and also had a time limit (usually thirty minutes) and a limit of 
attempts: these were especially useful in the flipped classroom modality. Despite this, some teachers did not 
want to adopt the flipped classroom modality because they feared that parents would do the homework instead 
of their children. Obviously, when students do exercises at home we have less control, but if there is a 
constructive discussion at home between students and family members for the resolution of the exercises, which 
is also welcomed. When designing exercises, we tried to create many different types. Not every question has a 
pop-up link: some suggest students to use the dictionary they want instead. Another typology of exercise did 
not require the consultation of an online dictionary as we integrated part of the dictionary entry in the body of 
the question. For example, in one exercise we inserted the modified dictionary entry “campo” from the Italian 
online dictionary De Mauro where we deleted a part of it and we also deleted the four labels from where they 
were placed. Students had to reintegrate them properly choosing among the different options proposed in the 
drop-down menu and associating for each definition the correct label associated to it. Students could choose 
between the following shortened definitions of four labels: FO fundamental, CO common usage, TS specialistic 
term, OB obsolete meaning. It is clear that, if we had inserted a pop-up link, students would have easily seen 
the labels associated with each item and the exercise would have become trivial copying. The goal of the 
exercise was instead to test sociolinguistic awareness of native speakers and to reach it the consultation of an 
online dictionary was not required at all. 


4.2 Are “Esplora-Students” Really Exploring Online Dictionaries? 


Students carried out 581 tests during the two editions of the project (the first one in the school year 17/18 and 
the second one in 18/19). In detail, 53 students took the “English level B1” test, 25 took the “German” test, 
244 took the “Italian: Grammar and Synonyms” test, 199 took the “Italian for the 9th grade” test, 60 took the 
“Ttalian for the 10th grade” test. The goal was to understand how much time each student spent consulting the 
online dictionary and if that influenced the performance of the test. At present, it is not possible for us to 
automatically track the activity of students while consulting the online dictionary, nor to understand whether 
the pop-up link has been opened, so we do not know how much time students actually spent consulting the 
dictionary or what exactly they were looking at. What we did was therefore to analyze data on response times. 
Students performed tests in a “protected environment” as they did them either at school supervised by teachers 
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and tutors or at home with some automatic restriction of the AAS. For this reason, we assumed that “taking 
more time in carrying out the test” implies “having spent more time in the consultation of the dictionary” and 
we tried to understand if there was a correlation between response times and success in the answers given, for 
each test performed. Data were analyzed with SPSS in order to understand to what extent the use of the AAS 
(that allowed the use of online dictionaries) helped the students. Statistically results are significant when the 
bilateral asymptotic significance of the chi-square test (the so-called “p-value”) is less than 0.05 and the ETA 
is greater than 0.3. Table 2 shows some general results regardless of the test done. These are statistically 
significant as the p-value is lower than 0.001 and ETA is equal to 0.323. Best results were obtained when 
students spent between 16 and 20 minutes taking the test. In this table (and in the next ones) there will always 
be the label “U = undefined time” as it is not rare that students start the test but do not close it correctly, losing 
data about the exact amount of time spent on the test. 


Table 2. Correlation between duration and performance of all 581 tests 


Duration Performance # of students % of students 
between 1 minute 52.56% 90 15,49% 
and 9 minutes 
between 10 minut 63.05% 99 17,04% 
and 15 minutes 
between 16 minut 71.08% 94 16,18% 
and 20 minutes 
between 21 minut 66.00% 100 17,21% 
and 24 minutes 
between 25 minut 66.05% 91 15,66% 
and 50 minutes 
undefined time 26.26% 107 18,42% 

Total 56.93% 581 100% 


Results of Table 2 do not take into consideration length and difficulty of tests. In order to analyze in a more 
critical way the data collected, we divided them into two different categories of “classes” according to the time 
spent in performing the test: 

e = Table 3 shows results divided in classes A, B, C, D, E, F, G, U where we tried to keep the number 
of minutes constant (about 5 minutes per class); 

e ~=Table 4 shows results divided in classes P, Q, R, S, T, U where we tried to keep the number of 
students constant (about 97 students per class regardless of the test). 

Both tables record for each test which “class of time” performed better and in specific which was the mean 
of the results of the mentioned class (e.g. in table 3 it is written that 199 students carried out the test “Italian 
for the 9th grade”, the best result was obtained by 38 of them who spent between 16 and 20 minutes to carried 
it out and on average scored 70.72%). 


Table 3. Classes with number of minutes constant 


Classes with constant number of minutes: about 5 minutes per class. 
A= 1-5 minutes, B = 6-10 minutes, C = 11-15 minutes, D = 16-20 minutes, E = 21-25 minutes, F = 26-35 minutes, G 
= 36-50 minutes, U = “undefined time” 


# oe # of student 
student , Performance of . pram ETA 
T Class with the i in the 
est s who fest oneforianied the mentioned mentened P value squared 
carried P class value 
it out class 
English level B1 53 E = 21-25 minutes 80.64% 2 <0.001 0.669 
Italian: 
Grammar and 244 D= 16-20 minutes 74.52% 33 <0.001 0.481 
Synonyms 
jae ihe 199 D=16-20 minutes 10.72% 38 <0.001 0.352 
ihe 60  D=16-20 minutes 58.83% 7 <0.001 0.702 
German 25 D = 16-20 minutes 79.80% 8 0.252 0.226 
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It turned out that, no matter the category of classes analyzed, the results found were the same. We have not 
considered results given by the “German” test (as it is not statistically significant) but we want to show it 
anyway in this paper. All the other results are statistically significant as they all have p-value <0, 001 and ETA 
value > 0.3. Table 3 shows that students who spent between 21 and 25 minutes had the best performance in the 
“English level B1” test, although there were only two of them. For all the other Italian tests, the best results 
were achieved by those students who spent between 16 and 20 minutes. 


Table 4. Classes with number of students constant 


Classes with constant number of students: about 97 students per class, regardless of the test. 
P = 1-9 minutes, Q = 10-15 minutes, R = 16-20 minutes, S = 21-24 minutes, T = 25-50 minutes, U = “undefined time” 
# of 


student : Performance of id odes ETA 
Class with the best ‘ in the 
Test s who itforiinive the mentioned maenGaned P value squared 
carried P class value 
it out class 
English level B1 53 S = 21-24 minutes 80.64% 2. <0.001 0.624 
Italian: 
Grammar and 244 R = 16-20 minutes 74.52% 33 <0.001 0.479 
Synonyms 
oe he 199 R= 16-20 minutes 10.72% 38 <0,001 0.343 
sen has 60  R=16-20 minutes 58.83% 7 <0.001 0.701 
German 25 R = 16-20 minutes 79.80% 8 0.252 0.226 


Results showed in Table 4 reflect the ones of Table 3 and they both show that: 

e for the 3 Italian tests, students who had a better performance were in the D or R class, that means 
it took them between 16 and 20 minutes to do the test; 

e for the English test, students who had a better performance were in the E or S class, that means it 
took them between 21 and 25 minutes. 

We can conclude that, for Italian tests Italian-speaking students were able to have a good performance even 
without spending too much time consulting the dictionary while for the English test consulting the dictionary 
made the difference, resulting in better results for those students who consulted it more. Moreover, exercises 
in these tests were specifically designed in such a way that they could not be correctly carried out without 
knowing how to use dictionary microstructure: for this reason, we can conclude that it is likely that students 
who performed better learned more how to use the online dictionary. 


5. CONCLUSION AND FURTHER STUDIES 


The integration of a VLE with an AAS for the consultation of online dictionaries allowed researchers to become 
“design aware” and students to become “meta-linguistically aware”. In fact, the AAS records all answers given, 
allowing more studies about how to improve both the content and the layout of exercises. The AAS also records 
the time spent on each test. Assuming that “taking more time in carrying out the test” implies “having spent 
more time in the consultation of the dictionary’, we concluded that, not surprisingly, for an Italian test 
Italian-speaking students were able to have a good performance even without spending too much time 
consulting the dictionary. On the contrary, for the English test, consulting the dictionary made the difference, 
resulting in better results for Italian speaking students who consulted it more. This also answers our first 
research question, that is how to increase the use and the circulation of online dictionaries among students: 
thanks to the project “Esplorare (con) i Dizionari Digital”, students were encouraged to use them and trained 
on how to, especially when foreign languages tests were involved. However, at present it is not possible for us 
to automatically track the activity of students while consulting the online dictionary, so a further study would 
be interesting in order to know whether the pop-up link has been opened or not, and which specific part of the 
online dictionary has been consulted; the eye-tracking technology could help with this. In addition, we had 
really positive feedback on the project from students and teachers who took part in it. For further studies, we 
want to ask them to fill a final questionnaire in order to see the increment of motivation and interest of students 
and to better measure the meta-cognitive impact of the project. In this way, we could add some qualitative 
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analysis to this purely quantitative one. “Esplorare (con) i Dizionari Digitali” showed us that development of 
competences for languages and STEM disciplines does not differ as much as it seems; thus, it would be 
interesting to fully exploit and implement the adaptive formative model of STEM disciplines to the teaching 
of languages, too. 
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